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HAND PATTERN SWITCH DEVICE 
Background of the . Invention 

Technical Field 

5 The present invention relates to a hand pattern switch 

device through which an operator inputs operation information 
to a controlled object, and more particularly, to a hand pattern 
switch device for enabling a driver to easily operate a 
vehicle -mounted equipment such as air conditioner equipment and 
10 audio equipment, without his/her driving being affected and 
without the need of touching an operation panel of the 
vehicle -mounted equipment. 

Related Art 

15 There has been proposed a technical art (for example 

JP-A-11-134090) to operate vehicle -mounted equipment such as 
air conditioner equipment and audio equipment without touching 
an operation panel of the vehicle -mounted equipment, in which 
an image of a body part (for example, left hand) of a driver 

20 is picked up by a camera and subject to pattern recognition to 
obtain Information that is used to operate the vehicle-mounted 
equipment . 

In another proposed technical art (for excimple JP-A- 
2001-216069), information used to operate vehicle-mounted 

25 equipment is acquired from a driver's gesture such as finger 
pattern and finger motion. 

This kind of art, realized by the pattern recognition to 
recognize a shape or pattern of a palm, hand, fingers, or the 
like from a picked-up image thereof or realized by the motion 

30 detection to detect a motion of a palm, hand, fingers, or the 
like by tracing its positional change, is called as a hand 
pattern switch device in the present specification. 
Hereinafter, palm, hand, fingers, etc., are collectively 
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referred to as "hand" unless otherwise particularly specified. 

As mentioned above, a variety of auxiliary equipment such 
as audio equipment , air conditioner equipment , etc . , are 
mounted in a vehicle. In addition, each equipment has various 
5 adjustment functions. For instance, vehicle-mounted audio 
equipment such as a radio broadcast receiver, tape player, CD 
player, etc., each have volume, channel selection, play, and 
stop functions. Air conditioner equipment has functions of 
adjusting temperature, wind volume, wind direction, etc. 
10 In both the above-mentioned prior art examples, however, 

a variety of hand patterns are to be used each of which specifies 
a corresponding one controlled object , and another hand pattern 
Is used to specify an amount of operation. Thus, an Individual 
driver Is required to memorize an extremely large number of hand 
15 patterns, and accordingly his/her driving may be hindered. 

On the other hand, a hand pattern switch device Is only 
capable of Inputting pieces of Inforroatlon each of which is, 
in general, uniquely determined in accordance with what pattern 
is formed by a hand or In what way a hand motion is performed, 
20 or the like. This poses a problem in properly adjusting 
respective functions of various switch-operation objects. 

Specifically, some controlled objects such as channel 
selection on radio broadcasting and wind amount are preferable 
to be subject to a large continuous /stepwise operation, whereas 
25 other controlled objects such as temperature and sound volume 
are preferable to be subject to a continuous fine operation. 

Thus, it is preferable that switch-operation information 
to be input from the hand pattern switch device is made 
changeable In accordance with for example the preference of a 
30 driver or the controlled object, so as to Improve operablllty 
of the switch device. 

It Is also Important to enable a driver to use a hand 
pattern switch device, without extra load that may hinder 
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his/her driving, when he/she inputs information to control the 
operation of vehicle-mounted equipment. Furthermore, it is of 
course preferable that the operation information can be input 
easily and appropriately in accordance with which vehicle - 
5 mounted equipment serving as a controlled object is to be 
operated. 

Summary of the Invention 

An object of the present invention is to provide a hand 

10 pattern switch device, which is excellent in operability, for 
enabling an operator such as driver to properly input operation 
information, with ease, to controlled objects through the 
switch device, which information is used to control operations 
of the controlled objects such as vehicle -mounted equipment, 

15 the hand pattern switch device being in particular suitable to 
input the operation information in a manner suited to each 
individual controlled object. 

Another object of the present invention is to provide a 
hand pattern switch device for enabling an operator to input 

20 operation information to controlled objects through the switch 
device, without causing an extra load on the operator. 

According to the present invention, there is provided a 
hand pattern switch device, which has image pickup means for 
picking up an image of a hand that is within a predetermined 

25 image pickup zone and in which a hand pattern is recognized from 
and a hand motion is detected from the picked-up image to obtain 
operation information for a controlled object. 

A hand pattern switch device, according to one aspect of 
the present invention, comprises controlled object setting 

30 means for selecting one of controlled objects in accordance with 
the recognized hand pattern and/or the detected hand motion, 
and detection mode changing means for changing a mode of 
detection of the operation information based on the hand motion 
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In dependence on which controlled object Is selected. 

The controlled object setting means may cyclically select 
one of the controlled objects each time a predetermined hand 
motion pattern is detected. The detecting mode changing means 
5 may include a moved distance detecting mode where a moved 
distance of the hand in the image pickup zone is detected and 
used as the operation information, a stop time detecting mode 
where a stop time of the hand at a stop position, to which the 
hand has been moved by a predetermined distance or more in the 

10 image pickup zone, is detected and used as the operation 

information, and detection mode setting means for setting at 
least one of the moved distance detecting mode and the stop time 
detecting mode. More preferably, the moved distance detecting 
mode is a mode in which the moved distance of the hand is detected 

15 in mult 1- stage fashion using a predetermined moved distance as 
a unit of detection. 

According to the above-mentioned hand pattern switch 
device, the mode of detection of operation information that is 
provided to controlled objects can be set properly for example 

20 in accordance with characteristics of controlled objects, the 
preference or habit of an operator such as driver, or the like. 
Thus , practical advantages can be achieved such that controlled 
objects, such as adjustment objects of a variety of 
vehicle -mounted equipment, can be effectively operated under 

25 simplified operations by an operator with reduced load. In 
particular, since the operation information detecting mode can 
be changed in accordance with controlled object, the operation 
information can advantageously be input to any controlled 
object. Irrespective whether the controlled object is 

30 preferable to be operated stepwise with a relatively large 
amount of operation or to be operated continuously with a 
relatively small amount of operation. 

According to another aspect of the present Invention, 
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there Is provided a vehicular hand pattern switch device. In 
which an Image pickup zone is at a location which is located 
laterally to a steering wheel of a vehicle and to which a driver 
who steers the steering wheel can extend his/her arm without 
5 changing a driving posture. 

The image pickup zone may be at least 50 mm apart from 
an outer periphery of the steering wheel, and may be a rectangle 
in shape and may have a size of about 600 mm in a fingertip 
direction and about 350 mm in a width direction of the driver's 

10 hand which is extended. The detection of the hand motion in 
the image pickup zone may include detecting a 10 mm to 70 mm 
displacement of a centrold position of the hand, determined from 
the plcked-up image, as a controlled object selection and/or 
an amount of operation of a controlled object concerned. 

15 According to the aforementioned vehicular hand pattern 

switch device, operation instruction based on hand pattern and 
hand motion can be reliably detected, without being affected 
by a driver' s hand motion or arm motion for a driving operation. 
This is practically advantageous, for example, in that 

20 operation information can properly be provided to a desired 
vehicle -mounted equipment in accordance with detection results . 
In particular, a hand motion, which is an imaginary operation 
performed by the driver with a reality while resting his/her 
arm on a rest arm, can be detected reliably. Thus, the driver's 

25 load at the time of operation can be advantageously reduced, 
for instance. 

In a vehicular hand pattern switch device according to 
still another aspect of this invention, image pickup means is 
disposed at a location at which the image pickup means can pick 
30 up an image of a zone to which a driver who steers a steering 
wheel of a vehicle can extend his/her arm without changing a 
driving posture. The image pickup means may be disposed at a 
celling of the vehicle. 
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With this hand pattern switch device, the driver is 
enabled to input operation information to the switch device, 
and to a controlled object concerned, with a reduced load. 

A hand pattern switch device according to a further aspect 
5 of this invention comprises controlled object selecting means 
for cyclically selecting one of controlled objects when a 
predetermined hand motion pattern is repeated, and operation 
amount changing means for changing an eunount of operation of 
the selected controlled object in accordance with the hand 
10 motion . 

The hand pattern switch device may include operation 
start recognizing means for recognizing that an operation of 
said device is started when it is determined that the recognized 
hand pattern corresponds to a predetermined first hand pattern, 

15 and the controlled object selecting means may select a 

controlled object in accordance with a motion of the hand whose 
recognized hand pattern corresponds to a predetermined second 
hand pattern, after recognizing that the operation of said 
device is started. The operation start recognizing means may 

20 include a confirming means for confirming that the hand of the 
first hand pattern does not grasp or touch a particular thing 
and then determining that the operation of said hand pattern 
switch device is started. Further, the first hand pattern may 
correspond to one that is formed when the hand grasps a thing, 

25 and the second hand pattern may correspond to one that is formed 
when the hand indicates a thing with its finger. The hand 
pattern switch device may include start promoting means for 
promoting start of the operation amount changing means when the 
recognized hand pattern corresponds to a predetermined third 

30 hand pattern and a detected position of the hand in the image 
pickup zone is not displaced for a predetermined time after the 
controlled object is selected. 

According to the aforementioned hand pattern switch 



device, a cyclic selection of controlled object can be made by 
an operator such as driver by, for example, repeatedly 
performing a hand motion pattern corresponding to a push 
operation of a button switch with his/her forefinger. Moreover 
for example, an amount of operation of a gear shift lever can 
be changed by performing, with the first hand pattern, a hand 
motion corresponding to an operation of moving the gear shift 
lever with a hand. In this regard, in the technical arts 
disclosed in JP-A-H-134090 and JP-A-2001-216069 , each 
individual finger pattern is assigned to a corresponding one 
controlled object, and another finger pattern is assigned to 
determination of amount of operation. For this reason, the 
driver must memorize a large number of finger patterns, and 
hence his/her driving may be hindered* 

Brief Description of the Drawings 
Fig. 1 is a view showing a general construction of a hand 
pattern switch device according to an embodiment of this 
invention; 

Fig. 2 is a view showing a finger image pickup zone in 
the hand pattern switch device shown in Fig. 1; 

Fig. 3 is a view showing a general procedure of operations 
of inputting switch- operation information in the hand pattern 
switch device; 

Fig. 4A is a view showing an example of finger pattern 
used to give instruction on switch operation to the hand pattern 
switch device; 

Fig. 4B is a view of another example of finger pattern; 

Fig. 4C is a view of a further example of finger pattern; 

Fig. 5 is view showing an excunple of controlled object 
selection by a push operation with fingers; 

Fig. 6 is a view showing an example of an input mode setting 
screen used to set Input modes corresponding to controlled 
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Objects; 

Fig. 7 is a flowchart showing part of processing 
procedures for switch-operation information input in the hand 
pattern switch device; 
5 Fig. 8 is a flowchart showing processing procedures that 

follow the procedures shown in Fig. 7; 

Fig. 9 is a flowchart showing procedures for detection 
of switch-operation information in a distance mode; 

Fig. 10 is a flowchart showing procedures for switch- 
10 operation information detection in a time mode; 

Fig. 11 is a flowchart showing procedures for switch- 
operation Information detection in a distance/ time mode; 

Fig. 12A is a graph showing a relation between amount of 
finger motion and frequency of an imaginary switch operations 
15 of a push button switch performed by drivers with a reality; 

Fig. 12B is a graph showing a relation between amount of 
finger motion and frequency of Imaginary switch operations of 
a lever switch performed by drivers with a reality; and 

Fig. 12C is a graph showing a relation between stop time 
20 and frequency of stoppages of imaginary switch operation 
performed by drivers with a realty. 

Detailed Description 
A hand pattern switch device according to an embodiment 

25 of this invention will be explained with reference to the 
drawings. In the following, palm, hand, fingers, etc. are 
collectively referred to as "hand," unless otherwise 
particularly specified. 
[Overall Construct Ion] 

30 Fig. 1 is a view of a general construction of essential 

part of the hand pattern switch device according to the present 
embodiment, showing a state around a driver's seat of a truck 
and functions of the switch device realized for example by a 
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microcomputer (ECU) and the like. Near the driver's seat, a 
steering wheel 1 steered by a driver, a combination switch (not 
shown), etc. are provided, whereas an operating section 2 for 
audio equipment, air conditioner equipment, etc, serving as 
5 controlled objects, is provided on a console panel. At a 
celling located above the driver's seat, a video camera 3 Is 
disposed for picking up an Image of fingers of a driver who 
extends his/her hand to an Image pickup zone located laterally 
to the steering wheel 1. The camera 3 Is comprised of a 

10 small-sized CCD camera or the like* 
[Camera Arrangement] 

As shown in Fig. 2, the Image pickup zone 3a of the camera 
3 located laterally to the steering wheel 1 Is at least 50 mm, 
preferably about 100 mm, apart from an outer periphery of the 

15 steering wheel 1 . And the image pickup zone 3a is rectangle 
In shape and has a size of about 600 mm in a fingertip direction 
of and about 350 mm in a width direction of the driver's hand 
extended laterally to the steering wheel 1. 

Since the image pickup zone 3a is set as mentioned above, 

20 the driver's hand does not fall within the image pickup zone 
3a when the driver holds the steering wheel 1 or operates the 
combination switch (not shown) provided at a steering column 
shaft, and the driver can move his/her fingers Into the image 
pickup zone 3a without largely moving the arm. In particular, 

25 the driver can extend his/her arm to the Image pickup zone 3a, 
without changing a driving posture, while resting the arm on 
an arm rest 5 that is provided laterally to the driver's seat. 
And the image pickup zone 3a Is located away from the operating 
section 2 for audio equipment, etc. 

30 With the above arrangement, a hand motion for a driving 

operation or for a direct operation of the operating section 
2 of the audio equipment, etc. is prevented from being 
erroneously detected as a switch operation. 
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If a gearshift lever (not shown) used to operate a 
transmission Is located In the Image pickup zone 3a set as 
described above, a pressure- sensitive sensor for example may 
be provided In the gearshift lever to make a detection as to 
5 whether the gearshift lever Is grasped by the driver. The 
provision of such sensor makes It possible to easily determine 
which of the gearshift lever or the hand pattern switch device 
Is operated by the driver ' s hand extended to the Image pickup 
zone 3a, whereby a driving operation is prevented from being 

10 erroneously detected as a switch operation. Alternatively, a 
height of driver's hand (I.e. , distance from the camera 3) may 
be detected by using a stereoscopic camera serving as the camera 
3, to detezrmlne whether the driver's hand, extended to the Image 
pickup zone 3a, operates the gearshift lever or Is present In 

15 a space above the gearshift lever. 

As mentioned above, the setting of the Image pickup zone 
3a Is made such that the driver can move his/her fingers to this 
zone, without largely moving the arm. Besides , the Image pickup 
zone Is set such that the driver can make a finger motion 

20 (displacement) for an Imaginary switch operation In this zone, 
with a reality. When the driver actually operates a switch of 
audio equipment, etc. , a reaction force Is produced, and hence 
the driver has a reality that he/she has made a switch operation. 
On the other hand. In an Imaginary switch operation, the driver 

25 simply moves his/her hand In a predetermined space, and no 
reaction force Is produced. Thus, the operator does not have 
a direct bodily sensation, but has a sensory Impression by way 
of his/her finger motion or hand motion for the Imaginary switch 
operation . 

30 In this connection, the present Inventors made 

experiments to ascertain how much hand motion Is required to 
enable an operator to have a sensory Impression that he/she has 
certainly made an Imaginary switch operation. Instead of 
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actually operating a push button switch once at a time or Instead 
of actually operating a shift lever In a multi-stage fashion. 

In each case, the hand motion is kept stopped for a while 
to indicate completion of the switch operation. Thus, the 
present inventors performed further experiments to ascertain 
how much stop time is required to enable an operator to have 
a sensory impression that he/she has completed a switch 
operation. 

In these experiments, eunounts of hand motion 
(displacement) and stop time required to provide an operator 
with sensory Impressions that he/she has made a switch operation 
and has completed such operation were repeatedly measured under 
the cooperation of operators who have a driver's license for 
big size vehicle. 

Figs. 12A-12C show results of the experiments. 
Specifically, Fig. 12A is a graph In which the frequency of 
imaginary switch operations is shown as a function of amount 
of hand motion (displacement) with which some operators felt 
that they had made an imaginary operation that corresponds to 
a depressing operation of a push button switch. This 
experimental result indicates that most drivers felt that they 
had made such a switch operation when moving their hand by 50 
mm, although the amount of hand motion (displacement) varies 
from 30 mm to 70 mm and the average amount is 45 mm. 

As for imaginary switch operations corresponding to a 
back and forth multl- stage operation of a shift lever, most 
drivers felt that, as shown in Fig. 12B, they had made a one-stage 
operation when moving their hand by 30 mm although the eunount 
of finger motion (displacement) varies from 10 mm to 40 mm and 
the average amount Is 23 mm. As for a stop time for which an 
operator stops his/her hand motion, following a predetermined 
amount of hand motion, to Indicate completion of a single switch 
operation, most drivers felt that they had completed the 
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operation when stopping their hand motion for 320 msec, as shown 
In Fig. 12C, The stop time was measured In units of 80 msec. 

As understood from the experimental results. In a case 
where a driver makes an imaginary switch operation in a natural 
5 state with his/her arm resting on an arm rest, the driver has 
a sensory Impression with a reality (click feeling) that he/she 
has made the operation when the hand motion is about 50 mm for 
the single push button switch operation and about 30 mm for the 
one-stage shift lever operation, and has a sensory Impression 

10 that he/she has completed the operation when the hand motion 
Is stopped for about 300 msec. 

The position and size of the Image pickup zone of the 
camera 3 are determined by taking Into account of not only the 
experimental results but also other conditions: a multi-stage 

15 lever switch Is typically operated back and forth In five 

stages; a reference position, at which a driver's hand extended 
laterally to the steering wheel 1 Is initially placed, varies 
depending on driver * s body shape and driving posture; a typical 
length from a wrist to a fingertip is about 200 mm; and a typical 

20 hand width is about 120 mm. As a consequence, the image pickup 
zone 3a is determined to be a rectangle in shape and to have 
a 600 mm length and a 350 mm width. 

By setting the image pickup zone in the above manner, a 
driver's hand, coming off the steering wheel 1 and then 

25 naturally moved without a sense of Incompatibility, can be 
captured without fall and without a hand motion for a driving 
operation being erroneously detected. It Is also possible to 
reliably grasp a hand motion for an imaginary switch operation 
In the Image pickup zone which Is limited In size, so that pattern 

30 recognition of a hand and detection of an amount of hand motion 
(deviation) can easily be made with relatively simplified Image 
processing. 

For the driver, he/she can perform a desired Imaginary 
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switch operation by simply moving his/her hand after forming 
a predetermined hand pattern, while extending his/her arm 
laterally to the steering wheel 1 without changing a driving 
posture and without directly touching the operating section 2 
5 for audio equipment, etc. 

This reduces a load of the driver performing a switch 
operation. In addition, since a hand motion and/or an arm 
motion for a driving operation cannot erroneously be detected 
as an imaginary switch operation, the driver can concentrate 
10 on driving without paying attention to the hand pattern switch 
device, and can, where required, give instruction for switch 
operation by simply moving his/her hand to the image pickup zone 
3a. 

[Data Processing Section] 

15 Basically, the microcomputer (ECU) of the hand pattern 

switch device performs the processing to recognize a driver's 
hand pattern or hand motion on the basis of an image picked up 
by the camera 3, and acquires operation information 
(switch-operation information) for a controlled object such as 

20 audio equipment or air conditioner equipment, based on results 
of the recognition . Thus , the hand pattern switch device serves , 
instead of the operating section 2, to provide switch-operation 
information to the audio equipment, air conditioner equipment, 
etc. 

25 More specifically, as shown mainly in Fig. 1, the hand 

pattern switch device comprises the camera 3 serving as image 
pickup means, the microcomputer (ECU) for acquiring switch- 
operation information from the image picked up by the camera 
3, and the like. 

30 The ECU comprises a binarlzation processing section 11 

for finally binarlzing an input image (hereinafter referred to 
simply as input image) which is input from the camera 3 to the 
microcomputer (ECU), so that background image components are 
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removed to extract a hand Image from the picked-up Image; a 
pattern recognition section 12 for recognizing a hand pattern 
from the hand image extracted by the binarization processing; 
a centroid detecting section 13 for determining a centroid 
5 position of a hand based on the recognized hand pattern; and 
an instructed-operation recognizing section 14 for recognizing 
a switch- operation information given by the driver by his/her 
hand pattern or hand motion, based on results of recognition 
by the pattern recognition section 12 and the centroid position 

10 of the hand detected by the centroid detecting section 13 . 
This instructed-operation recognizing section 14 
generally comprises a pattern determination section 16 for 
determining and identifying a hand pattern, referring to hand 
patterns registered beforehand in a memory 15, a displacement 

15 detecting section 17 for tracing a motion of centroid position 
of a hand to detect a displacement of the hand, and a timer 18 
for monitoring a time that has elapsed with hand motion. 
Preferably, a palm portion of the hand is subject to the centroid 
position detection, moved amount monitoring, and elapsed time 

20 monitoring. 

The instructed-operation recognizing section 14 is 
arranged to determine information on predetermined switch- 
operation information specified by the driver's hand pattern 
and/or hand motion, and output the information to the audio 

25 equipment, air conditioner equipment, or the like. 

Further, the instructed-operation recognizing section 14 
is provided with a guidance section 19 that provides a 
predetermined guidance (announcement) to the driver according 
to results of the aforementioned determination, etc. The 

30 driver is notified of the guidance in the form of a speech message 
that specifies for example the audio equipment or air 
conditioner equipment (controlled object equipment), or 
volume /channel setting, wind volume /temperature, or the like 
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(controlled object function), or In the form of confirmation 
sound such as beep tone or blip tone that identifies a switch 
operation having been made. 

One of features of the hand pattern switch device 
constructed as described above is to change input mode for 
inputting switch-operation information to controlled object in 
accordance with which one of the controlled objects is selected, 
as shown for example by switch operations in Fig. 3. To this 
end, three types of input modes, namely, distance mode, time 
mode, and distance/ time mode, are provided so that an 
appropriate one of which may be selected depending on the 
selected controlled object (vehicle -mounted equipment). 
[Outline of Hand Pattern Switch Operation] 

Next, the switch operation will be described. 

In the hand pattern switch device, finger patterns shown 
in Figs. 4A-4C are used as hand patterns each specify a 
corresponding switch operation. Specifically, a clenched- 
fist type pattern (hand pattern 1) shown in Fig. 4A, a 
f oref inger-up type pattern (hand pattern 2) shown in Fig. 4B, 
and a thumb-up type pattern (hand pattern 3) shown in Fig. 4C 
are used. 

The f oref inger-up type pattern (hand pattern 2) is 
intended to express image of depressing a switch such as a push 
button. The clenched-f ist type pattern (hand pattern 1) 
describes image of grasping a lever switch, such as a shift lever 
used to change the transmission stage of an automatic 
transmission , and shifting it back and forth. The thumb-up type 
pattern (hand pattern 3) describes image of standby instruction. 
These hand patterns are registered beforehand in the memory 15 
in the form of data representing their features of patterns; 
The patterns are used to make a hand pattern determination in 
which a degree of conformity (similarity) is calculated between 
a hand pattern recognized from an input image and each 
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Individual pattern registered In the memory 15. 

When the driver grasps the steering wheel 1 to drive the 
vehicle, driver's hands fall outside the Image pickup zone 3a 
as shown by operation state Al In Fig. 3. The Input Image at 
5 that time only Includes background image components that will 
be removed, so that the hand pattern switch device does not 
operate. On the other hand, when a driver's hand comes off the 
steering wheel 1 and enters the image pickup zone 3 as shown 
by operation state A2, the processing for hand pattern 

10 recognition starts. At that time, it is preferable that the 
driver clenches his/her fist to form hand pattern 1, i.e., 
clenched-f 1st pattern, thereby indicating his/her intention to 
start switch -operation information input. 

Then, a determination is made as to whether the hand 

15 pattern is changed to the f oref inger-up pattern (hand pattern 
2) to make an operation of depressing a switch. If the 
f oref inger-up pattern (hand pattern 2) is detected, detection 
of a hand motion starts, assuming that a depressing operation 
of a selected switch is repeated as shown in operation state 

20 A3 in order to selectively specify a desired function/equipment . 
Specifically, a change in the centrold position of the hand 
(i.e. , a hand motion) is detected to thereby detect a back and 
forth motion of fingers with the f oref inger-up pattern, whereby 
push operations are detected. 

25 In this connection, a control algorithm may be used, in 

which, each time a push operation is detected, a selectable 
controlled object is specified in a cycle from sound volume to 
channel, to temperature, and to wind volume, etc. , as shown by 
way of example in Fig. 5. 

30 In addition, each time a push operation is detected, it 

is useful to output confirmation sound such as blip tone or a 
sound message to notify the driver of the controlled object that 
can be selected or is selected. With such guidance, the driver 
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can recognize a state of switch operation while concentrating 
on driving, without the need of visually confirming the switch 
operation. 

When the desired controlled object is selected, the hand 
5 pattern is changed to a different pattern, e.g. , to the thumb-up 
pattern (hand pattern 3 ) as shown for example in operation state 
A4 , to indicate an intention of completing the selection of 
function and indicate an Intention to start to input a switch 
operation amount. At this time, it is preferable that 

10 confirmation sound such as beep tone different from the 
aforesaid one is produced. 

Subsequently, the hand pattern is changed to the 
clenched-f 1st pattern (hand pattern 1) as shown in operation 
state A5, and the hand with hand pattern 1 is moved back and 

15 forth to input switch -operation information. At this time, a 
hand motion with hand pattern 1 is detected based on a deviation 
of the centroid position of the hand or the like, and 
switch- operation Information corresponding to such hand motion 
is output . 

20 Specifically, at the time of inputting the switch- 

operation information, information content to be detected is 
changed in accordance with the controlled object selectively 
specified by the push operation. 

In general, controlled objects can be classified into two. 

25 One is that whose amount of control (switch-operation 

Information) is preferably changed in a multl- stage fashion 
(stepwise) with a large Increment of change, and another one 
Is that which Is preferable to be changed continuously and 
finely with a small Increment of change. 

30 In this regard, depending on the selected controlled 

object, a corresponding one of the following input modes is 
selectively set: a distance mode (operation state A6a) where 
an amount of motion (displacement) of the hand is Input as the 
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switch-operation Information; a time mode (operation state A6b) 
where the switch- operation Information Is determined based on 
a stop time of the hand which has until then been moved by a 
predetermined distance or more; and a distance/ time mode 
5 (operation state A6c) where these two modes are used in 
combination . 

The distance mode is a mode for outputting the 
switch-operation information that varies in a multi-stage 
fashion in accordance with a moved distance of the hand from 
10 a reference position, and is suitable for channel selection on 
radio broadcasting or for selection of wind amount/wind 
direction of air conditioner equipment. 

The time mode Is a mode for outputting the switch- 
operation information varying depending on a stop time of the 
15 hand that has until then been displaced from a reference 

position, and is suitable for example for adjustment of sound 
volume of audio equipment and for adjustment of temperature of 
air conditioner equipment. 

The distance/time mode is a mode in which the swltch- 
20 operation information determined according to an amount of hand 
motion is output when the hand is moved slightly, whereas the 
Information determined according to a stop time of the hand at 
a stop position is output when the hand has been moved by a 
predetermined distance or more to the stop position. 
25 The distance/time mode is suitable for example for 

controlled objects that are subject to a fine adjustment after 
being roughly adjusted. 

In accordance with the input mode (distance mode or time 
mode or distance/ time mode) selectively set according to the 
30 controlled object, the Instructed-operatlon recognizing 

section 14 traces a hand motion in the image pickup region 3a, 
measures an amount of hand motion (amount and direction of 
displacement) or a stop time of the hand which stops moving. 
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and outputs the switch- operation information that varies 
depending on measurement results. 

A relation between various controlled objects in switch 
operation and input modes and/or information content (such as 
distance, time) to be detected in the detection of switch 
operation amount may be fixedly set in advance. Alternatively, 
the relation, distance, time, etc. may be made changeable in 
accordance with the preference of the switch operator (driver) , 
etc. by using for example a control display screen 14a shown 
in Fig. 6 that is provided in the instructed- operation 
recognizing section 14. 

More specifically, which input mode should be used to 
input the switch- operation Information may selectively be set 
for each individual controlled object. And the displacement 
(moved distance) of hand, measurement time, etc., serving as 
detection unit, may variably be set. 
[ Flowohart s ] 

(1) Initial Determination 

Figs. 7-11 show a general processing procedure performed 
by the instructed-operation recognizing section 14 to realize 
the aforementioned processing functions. 

In the processing procedure, upon start of operation of 
the hand pattern switch device, initial system setting is 
performed that includes setting of an image pickup zone 
(recognition zone) of the ceunera 3, setting of a threshold value 
for blnarization processing, etc. (step SI). 

Then, initialization as for the detection of switch- 
operation information is performed, in which parameter values 
used for hand pattern recognition and hand motion measurement 
are cleared, and a control flag is set to a value of 0 (step 
S2) . 

After completion of the initialization, a hand pattern 
in an input image is recognized (step S3). 
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Then, a determination Is made as to whether the recognized 
hand pattern corresponds to hand pattern 1, 2 or 3 (step S4). 

If the hand pattern corresponds to hand pattern 2 of 
f oref inger-up type, which hand pattern was detected In the 
5 preceding cycle is determined (step S5). 

If the preceding hand pattern is other than hand pattern 
2 of f oref inger-up type, it is determined based on the currently 
detected hand pattern 2 that a function selection mode is set, 
and a hand centrold position detected at that time is registered 
10 as a reference position A (step S6). 

Then, a parameter B' representing a displacement of the 
hand detected in the preceding cycle and used to manage the moved 
distance of the hand is set to a value of 0 (step S7). 

Then, the preceding hand pattern is renewed to hand 
15 pattern 2 (step S8). 

VHiereupon, the processing of step S3 and subsequent steps 
is repeatedly executed* 

On the other hand, if it is determined at step S5 that 
the preceding hand pattern is the f oref inger-up type hand 
20 pattern 2, a deviation (moved amount) B of a hand centrold 
position detected at that time from the reference position A 
is determined (step S9). 

Next, a further determination is made as to whether the 
following two conditions are satisfied. A first one is that 
25 the parameter B' is less than a determination threshold value 
HI, and a second one is that a currently detected displacement 
(moved amount) B exceeds the predetermined detection threshold 
value HI (step SIO), 

If these two conditions are satisfied, it is determined 
30 that a push operation in hand pattern 2 is made for function 
selection, so that a one-stage change In controlled object is 
performed (step Sll). 

Then, the preceding hand pattern Is renewed to hand 
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pattern 2 (step S8) , and the processing of step S3 and subsequent 
steps is repeatedly executed. 

If the two conditions are not satisfied at step SIO for 
the reason for example that the displacement B does not exceed 
5 the predetermined determination threshold value Hi, the 
displacement B Is set as the parameter B' (step S12). 

Then, the processing starting from step S3 Is repeatedly 
executed. 

As a result of the processing of steps S3-S8 being executed, 
10 when a push operation Is made by a predetermined move amount 
with hand pattern 2, the function selection is made to 
selectively set the controlled object concerned. 

On the other hand. If it is determined at step S4 that 
the recognized hand pattern corresponds to hand pattern 1 or 
15 3, a further determination is made as to which of hand patterns 
1 and 3 corresponds to the recognized hand pattern, as shown 
in Fig. 8 (step S13) . 

If the hand pattern corresponds to hand pattern 3, it is 
determined that an operation await state is specified, so that 
20 a hand centrold position detected at that time is registered 
as a reference position C (step S14). 

At the scune time, a control parameter D' used for moved 
distance measurement, mentioned later, is set to a value of 0 
(step S15) . 

25 Subsequently, the preceding hand pattern is renewed to 

hand pattern 3 (step S16) , and the processing starting from step 

S3 is repeatedly executed. 

On the other hand, if it is determined at step S13 that 

the recognized hand pattern corresponds to hand pattern 1 , the 
30 hand pattern detected in the preceding cycle is determined (step 

S17) . 

If the preceding hand pattern corresponds to hand pattern 
3, it is determined that the setting of operation information 
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Input mode Is performed, so that the control flag Is set to 1 
and parameters n and t for moved distance measurement and stop 
time measurement are set to 0 (step S18). 

If the preceding hand pattern Is the clenched-f ist type 
5 hand pattern 1, whether the control flag Is at 1 Is determined 
to thereby determine whether an operation Information Input 
mode has already been set (step S19). 

If the control flag Is at 1, a distance D between a hand 
centrold position detected at that time and the reference 
10 position C Is detected (step S20). 

In accordance with the controlled object having been 
selected by the function selection, a determination Is made as 
to which one of the three Input modes should be used for the 
detection of switch- operation Information (step S21). 
15 Whereupon, the processing of detection of switch- 

operation Information Is performed In accordance with the 
determined Input mode. 

If the preceding hand pattern does not correspond to hand 
pattern 1 or 3 (step S17), or If the control flag Is not at 1 
20 (step S19) , the processing procedure starting from the Initial 
setting at step S2 is repeatedly executed. 

As a result of this processing. Instruction Is provided 
to start the Input of switch-operation Information. This 
corresponds to operation state A2. 
25 (11) Distance Mode 

As Shown by way of example In Fig. 9, In the processing 
to detect switch-operation Information In the distance mode, 
a distance D from the reference position C Is compared with a 
value of control parameter D ' ( step S22 ) , and In accordance with 
30 a relation In size therebetween, whether the hand Is advancing 
or retreating Is determined. 

When the hand Is being advanced, whether the moved 
distance from the preceding cycle to the present cycle exceeds 
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a predetermined measurement unit distance Is determined. In 
other words, whether the later-mentioned condition 1 Is 
satisfied Is determined (step S23). 

If the measurement unit distance Is exceeded, the control 
5 parameter n is Incremented (step S24). 

Then, a one-stage Increase Is made In the setting of 
controlled object (switch-operation Information) (step S25), 
Conversely, if the hand is being retreated, whether the 
moved distance from the preceding cycle to the present cycle 
10 exceeds the predetermined measurement unit distance is 
determined. In other words, whether the later-mentioned 
condition 2 is satisfied is determined (step S26}. 

When the moved distance exceeds the measurement unit 
distance, the control pareuneter n is decremented (step S27). 
15 Then, the setting (switch-operation Information) is 

decreased by one stage (step S28). 

The condition 1 is satisfied when a difference between 
distances D and C (moved amount) exceeds a moved distance 
corresponding to a predetermined number of steps that is 
20 variably set by the control parameter n, but a difference 

between distances D' and C (moved amount) is less than the moved 
distance corresponding to the predetermined number of steps 
variably set by the control parameter n. 

In other words , the condition 1 is used to determine 
25 whether the hand has moved by the predetermined measurement unit 
distance or more that is determined by the predetermined ntimber 
of steps. The condition 2 is similar to the condition 1, but 
used for a case where the hand moves in the opposite direction 
( retreat ) . 

30 After the switch- operation information corresponding to 

the moved distance of the hand has been obtained in the above 
manner, the distance D is registered as the control parameter 
I>' , whereby the determination reference is renewed (step S29) . 



24 



Then, hand pattern 1 Is registered as the preceding hand 
pattern (step S30). 

Whereupon, the processing from step S3 is resumed. 

By repeatedly executing the processing, the switch- 
5 operation information depending on the moved amount of the hand 
is output step by step. 
(Ill) Time Mode 

On the other hand, if the time mode Is set, the detection 
processing is executed In accordance with procedures shown by 
10 way of example In Fig. 10. In this case, whether the moved 
distance D has exceeded a third threshold value H3 Is determined 
(step S31) . 

If the third threshold value H3 is not reached, a time 
parameter t Is set to 0 (step S32). 
15 Then, hand pattern 1 Is registered as the preceding hand 

pattern (step S33). 

Whereupon the processing starting from step S3 Is 
resiimed. 

If It is determined at step S31 that the moved distance 
20 D exceeds the third threshold value H3, the time parameter t 
is counted up (step S34). 

And a determination is made as to whether the counted-up 
time parameter t reaches a reference time Tl (step S35). 

If the time parameter t equal to or larger than Tl is 
25 determined, the setting ( switch- operation information) Is 
Increased by one stage (step 835). 

Then, the time parameter t is reset to 0 (step S37). 
Further, hand pattern 1 is registered as the preceding 
hand pattern (step S33). 
30 Whereupon, the processing from step S33 is resumed. 

If It is determined at step S31 that the moved distance 
D exceeds the third threshold value H3 In the opposite direction, 
the time parameter t is counted up (step S38). 
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Then, whether the counted-up time parameter t reaches the 
reference time Tl is determined (step S39). 

When the time pareimeter t equal to or larger than Tl is 
determined, the setting ( switch- operation information) is 
5 decreased by one stage (step S40), 

Then, the time parameter t is reset to 0 (step S41). 

Further, hand pattern 1 is registered as the preceding 
hand pattern (step S33). 

Whereupon, the processing from step S3 is resumed. 
10 By means of such a series of processing, when the hand 

has moved by a predetermined distance and stops there, the 
switch-operation information is output in accordance with the 
stop time* 

(iv) Distance/time Mode 

15 The distance/ time mode is executed, while selectively 

using the processing in the distance mode or the processing in 
the time mode in accordance with the moved distance of the hand. 

Specifically, as shown in Fig. 11, a value of the control 
parameter n is determined (step S42). 

20 Depending whether the pareimeter value n falls within a 

range from -2 to 2, or is equal to or larger than 3, or equal 
to or less than -3, the processing in the distance mode from 
step S22a to step S33a or the processing in the time mode from 
step S24a to S41a is executed. 

25 The processing from step S22a to step S33a corresponds 

to the processing from step S22 to step S33. The processing 
from step S24a to step S41a corresponds to the processing from 
step S24 to step S41. 

According to the hand pattern switch device having the 

30 above described functions, it is possible to easily and 

effectively select equipment to be controlled, selectively set 
the input mode suitable for the selected controlled object, and 
input the switch-operation information to the selected 
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controlled object. 

Thus, an amount of switch operation suitable to the 
controlled object can be input easily both in a case where wind 
amount or wind direction is selected as the controlled object 
5 which is set preferably by providing a relatively large 

displacement, and in a case where sound volume or temperature 
is selected which is preferably subject to a fine adjustment 
that is performed by providing a relatively small displacement. 
In particular, by setting the input mode beforehand 
10 depending on the controlled object, practical advantages can 
be attained such that appropriate switch- operation information 
can be easily input according to the intent of operation. 

By variably setting the unit distance and unit time for 
detection, etc., the setting can be made in a manner suitable 
15 for the body shape, preference, habit of the switch operator 
(driver) . 

Also, the setting suitable for equipment characteristics 
of controlled objects can also be achieved. This makes it 
possible to realize a hand pattern switch device capable of 

20 reducing operation load and easy to operate. 

Furthermore, according to the hand pattern switch device 
constructed as mentioned above, switch-operation instruction 
based on hand pattern and hand motion can be detected easily, 
effectively and reliably, without being affected by a hand 

25 motion for a driving operation, and in accordance with results 
of the detection, switch- operation information can be provided 
properly to the desired vehicle -mounted equipment. In 
particularly, the operation load to the driver can be reduced 
since the image pickup zone 3a in which an image of the hand 

30 to give the switch-operation instruction is captured is located 
laterally to the steering wheel 1 within a range to which the 
driver can naturally extend his/her arm without changing a 
driving position. Therefore, practical advantages can be 
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achieved such that the driver Is enabled to easily give an 
instruction to input the switch- operation information through 
the hand pattern switch device in much the same way that he/she 
directly operates the operating section 2 of audio equipment, 
5 etc. 

The present invention is not limited to the foregoing 
embodiment. In the embodiment, explanations have been given 
under the assumption that this invention is applied to a 
right -steering-wheel vehicle, but it is of course applicable 

10 to a left-steering-wheel vehicle. This invention is also 

applicable to an ordinary passenger car other than a large- 
sized car such as truck. As for the controlled object, 
expansions can be made to operation of navigation device, wiper 
on/off control, adjustment of interval of wiper operation, side 

15 mirror open/close control, etc. 

As for the size of the image pickup zone 3a, it may be 
variably set, utilizing a zooming function of the camera 3, for 
instance. Further, the position of the image pickup zone 3a 
may be subject to fine adjustment so as to meet the habit of 

20 individual driver, etc. To this end, the mount angle of the 
camera 3 may be adjusted. 

Furthermore, the switch-operation information input mode 
is not limited to the aforementioned three types, but an 
additional mode may be provided in which a hand pattern per se 

25 is input as switch- operation information. In the embodiment, 
sound volume, channel, temperature, wind eimount are directly 
selected as controlled objects. Alternatively, a controlled 
object equipment such as radio broadcast receiver, tape player, 
CD player, air conditioner, or the like may be specified. Then, 

30 a controlled object may be specified such as sound volume, 
channel, play, or stop function of audio equipment, or wind 
eunount, wind direction, or temperature function of air 
conditioner equipment . 
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In this case, the manner of detection of switch-operation 
Information may be variably set In accordance with the finally 
specified controlled object. 

In other respects, the present Invention may be modified 
variously, without departing from the scope of invention. 



